Migrate CI build-doc to Meson #39973

tobiasdiez · 2025-04-19T09:53:04Z

Make it possible to build the documentation with meson via

meson compile -C builddir doc-html

For this a couple of changes was necessary in the docbuilder:

Make sage_docbuild independent of sage so that meson can use sage_docbuild during config time to construct all the docbuild targets (otherwise one needs to first install sage, and then could configure the docbuild)
Properly handle input and output dirs for the docbuild, without relying on magic sage env variables

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

user202729 · 2025-04-19T10:16:41Z

Looks like some file permission goes wrong? (symbolic link get converted to normal file?)

user202729 · 2025-04-19T10:18:41Z

src/sage_docbuild/builders.py

+    docs = [doc[1] for doc in sorted(documents)]
+    # Put the bibliography first, because it needs to be built first:
+    docs.remove(Path('reference/references'))
+    docs.insert(0, Path('reference/references'))


Why is such a large change necessary in this file?

To be honest, I got carried a way a bit: the strictly necessary change is to remove mentions of import sage, then I used the chance to use pathlib, and to catch all those paths vs strings issues added typing info. Oh, and there were a few old deprecated stuff in there which was easier to remove than to migrate to pathlib 🦊

…tory-reference

github-actions · 2025-05-17T03:51:02Z

Documentation preview for this PR (built with commit 1c65a0b; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

tobiasdiez · 2025-05-17T10:28:32Z

Doc build is working now, and the changes relative to the last develop build seem to be minimal and mostly due to using a newer sphinx version now.

Marking it as blocker since currently the docbuild CI for other PRs is completely failing.

dimpase · 2025-05-17T15:56:44Z

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:1

fails while building docs the old way - is it normal?

tobiasdiez · 2025-05-17T16:40:21Z

It fails with

ImportError: libSingular-4.4.0.so: cannot open shared object file: No such file or directory

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:3982 so this is the same problem as for the other PRs.

dimpase · 2025-05-17T23:56:18Z

no, my question was - why does that CI run build docs the old way?

tobiasdiez · 2025-05-19T05:46:35Z

But because it's now marked as changed, the doctest gets explicitly invoked on it.

Hm, the doctest framework has a mechanism to mark something as not tested. (See https://github.com/sagemath/sage/pull/39102/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721R123 ). Maybe it is also applicable here?

Thanks, I was not aware of this mechanism!

I think it's the same issue as #39467 again.
either it is supported, then the code to test it should remain in the code base

I think the situation here is a bit different. The doc build ci workflow is mostly there to provide devs a preview of the doc changes; it's purpose is less to test the docbuilding itself (of course it's testing this as well). Also the meson and make scripts just invoke the sage docbuilder, which is doing most of the work - so I don't think it's worth to test both ways.

But yeah, it's a complicated question on how to best migrate completely to meson, also with respect to the required doc changes. On one hand, it's a lot of additional work to maintain the two systems in parallel (including docs and ci for both) while on the other hand you would want to make the transition as smooth as possible. I'm hoping that we can soft-deprecate the make interface in the next release cycle, which would be then a natural point to revise and migrate more of the documentation to meson. I'm open and happy about other opinions on how this migration should look like.

user202729 · 2025-05-19T10:35:55Z

Thanks, I was not aware of this mechanism!

do you plan to fix this later or just ignore it for this pull request?

tobiasdiez · 2025-05-19T12:59:53Z

Thanks, I was not aware of this mechanism!

do you plan to fix this later or just ignore it for this pull request?

Just wanted to add it to this PR, only to realize that it's not going to be that easy: the new workflow is just using ordinary git commands to stage all changed files (relative to develop), and then calls sage test --new. Probably can be fixed by committing changes to certain directories again manually; but that's definitely something for another PR.

dimpase · 2025-05-19T15:45:49Z

@user202729 - are you happy enough here now to revert this to positive review?

user202729 · 2025-05-19T19:50:41Z

Actually:

maybe change/append the PR title to "Migrate CI build-doc to meson"?
currently build-doc-pdf is not tested at all? What's the plan for that?

user202729 · 2025-05-19T20:02:43Z

Also the meson and make scripts just invoke the sage docbuilder, which is doing most of the work - so I don't think it's worth to test both ways.

Actually, the test is exactly doing its jobs, no?

In other words: is #40120 just an issue with the CI, or does an actual user using sage-the-distro (since that isn't completely deprecated yet we ought to assume some user exists…) trying to build doc that way would encounter the same error? The latter appears more likely? (I'm not using sage-the-distro myself)

In that case if the CI infrastructure is removed how do we know if a similar breakage won't happen again with some other package upgrade?

tobiasdiez · 2025-05-20T02:01:00Z

Actually:

* maybe change/append the PR title to "Migrate CI build-doc to meson"?

Done.

* currently build-doc-pdf is not tested at all? What's the plan for that?

I don't have a plan for this atm. To be honest, I don't even see the point of building a pdf of the docs in 2025. Having a good online documentation should suffice.

In other words: is #40120 just an issue with the CI, or does an actual user using sage-the-distro (since that isn't completely deprecated yet we ought to assume some user exists…) trying to build doc that way would encounter the same error?

The error is actually not related to the docs build. What happened is that the cython module was compiled with Singular x. Then Singular was updated to version y. The problem is that the CI/sage-the-distro doesn't recognize that the version of Singular changed and the cython module needs to be compiled again to link against Singular y and not x. So the underlying issue is with the caching of the CI/sage-the-distro.

user202729 · 2025-05-20T05:26:50Z

So the underlying issue is with the caching of the CI/sage-the-distro.

Indeed, I also guessed that. But isn't the CI caching the exact same mechanism as the normal caching anyway (e.g. an user of sage-the-distro originally use 4.4.0, then git pull to upgrade it to 4.4.1, will face the same error)

Anyway… can we somehow delete the CI cache to fix test-long for now?

I don't even see the point of building a pdf of the docs in 2025

In some rare cases, this catches some LaTeX errors… as a fallback if the author forget to check the documentation diff.

(Also, some HTML pages are extremely large that it lags my browser, it might be beneficial for someone to look at the PDF, because it's paginated it should render at reasonable speed)

user202729 · 2025-05-24T16:20:10Z

@vbraun What's the issue?

tobiasdiez · 2025-05-27T03:44:10Z

@vbraun is it because of http://build.sagemath.org/#/builders/47/builds/124? This indeed fails for the docs, but the full error points more to the m4ri update as the source.

vbraun · 2025-05-28T22:52:34Z

Does pdf docbuild work? I don't remember what the error was, but pretty sure the merge script failed because of that.

And yes, the point of running TeX is to verify that the equation are valid markup.

dimpase · 2025-05-29T01:27:41Z

@vbraun there are no changes to anything TeX-typesettable here.

user202729 · 2025-05-29T06:36:43Z

Does pdf docbuild work? I don't remember what the error was, but pretty sure the merge script failed because of that.

And yes, the point of running TeX is to verify that the equation are valid markup.

Not on the CI at the moment, but that was broken by a combination of previous pull requests. My understanding (best guess?) of the situation is explained in #40157 .

I'm not sure if build-doc-pdf depends on build-doc in some way however. A quick reading shows no dependency.

vbraun · 2025-05-29T07:35:19Z

----------------------------------------------------------------------
sage -t --long --warn-long 30.0 --random-seed=123 src/sage_docbuild/builders.py  # 16 doctests failed
----------------------------------------------------------------------

dimpase · 2025-05-29T17:19:19Z

I am trying to run meson compile -C builddir doc-html (after doing meson install) and I am getting errors
such as the following

[2/189] Generating src/doc/doc-inventory-reference-references with a custom command
[reference] Configuration error!
[reference] Versions
[reference] ========
[reference] * Platform:         linux; (Linux-6.13.4-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_Ultra_7_165U-with-glibc2.41)
[reference] * Python version:   3.13.3 (CPython)
[reference] * Sphinx version:   8.2.3
[reference] * Docutils version: 0.21.2
[reference] * Jinja2 version:   3.1.6
[reference] * Pygments version: 2.19.1
[reference] Last Messages
[reference] =============
[reference] None.
[reference] Loaded Extensions
[reference] =================
[reference] None.
[reference] Traceback
[reference] =========
[reference]       File "/usr/lib/python3.13/site-packages/sphinx/config.py", line 616, in eval_config_file
[reference]         raise ConfigError(msg % traceback.format_exc()) from exc
[reference]     sphinx.errors.ConfigError: There is a programmable error in your configuration file:
[reference]     Traceback (most recent call last):
[reference]       File "/usr/lib/python3.13/site-packages/sphinx/config.py", line 601, in eval_config_file
[reference]         exec(code, namespace)  # NoQA: S102
[reference]         ~~~~^^^^^^^^^^^^^^^^^
[reference]       File "/home/dima/software/sage-src/src/doc/en/reference/references/conf.py", line 19, in <module>
[reference]         for tag in feature_tags():
[reference]                    ^^^^^^^^^^^^
[reference]     NameError: name 'feature_tags' is not defined
[reference] The full traceback has been saved in:

which seem to indicate that no sphinx extensions are loaded. Why?

dimpase · 2025-05-29T23:32:33Z

@tobiasdiez : what is feature_tags It's something not defined in Sage sources

user202729 reviewed Apr 19, 2025

View reviewed changes

Meson: add building of documentation as target

10fea7f

tobiasdiez force-pushed the meson-docs branch from c90e95d to 10fea7f Compare April 19, 2025 10:27

tobiasdiez added 11 commits April 20, 2025 16:09

Fix meson doc build

d7c3b6e

Fix various issues

a805a97

Fix ci

e881cf1

Fix path to sage_docbuild

af4cb32

Add shell specification for storing old documentation

0470c2f

Try with editable install

199be4e

Fix directory creation

dd44ea7

add copybutton conda pkg

5ee3c51

Work with absolute paths everywhere

7550538

Add dependency for 'reference_top' on other reference docs

230fd8f

Exclude 'reference_top' from the list of other documents in doc-inven…

ee77f03

…tory-reference

tobiasdiez mentioned this pull request May 10, 2025

instaling sage cschwan/sage-on-gentoo#804

Open

tobiasdiez added 4 commits May 16, 2025 14:19

Merge remote-tracking branch 'origin/develop' into meson-docs

b07c0a6

Improve dependency handling

f71e282

Add dependency of 'top' docs on all other references

327a7bb

Use ccache to build sage

f6d186b

Don't create neasted doc directories

5a3c014

tobiasdiez marked this pull request as ready for review May 17, 2025 10:23

github-actions bot added the s: needs review label May 17, 2025

tobiasdiez requested a review from dimpase May 17, 2025 10:28

tobiasdiez added the p: blocker / 1 label May 17, 2025

user202729 mentioned this pull request May 18, 2025

Test failure in test-long etc. because of singular #40120

Closed

user202729 added s: positive review and removed s: needs work labels May 19, 2025

user202729 added s: needs info and removed s: positive review labels May 19, 2025

tobiasdiez changed the title ~~Meson: add building of documentation as target~~ Migrate CI build-doc to Meson May 20, 2025

user202729 added s: positive review and removed s: needs info labels May 20, 2025

vbraun added the s: needs work label May 23, 2025

user202729 mentioned this pull request May 26, 2025

Add back ubuntu-noble-standard to ci-linux #40157

Merged

5 tasks

dimpase approved these changes May 29, 2025

View reviewed changes

dimpase removed the s: positive review label May 29, 2025

user202729 mentioned this pull request May 31, 2025

Make preparser can handle large(4301+ digits) integers #40178

Open

5 tasks

Merge branch 'develop' into meson-docs

1c65a0b

Uh oh!

Migrate CI build-doc to Meson #39973

Are you sure you want to change the base?

Migrate CI build-doc to Meson #39973

Uh oh!

Conversation

tobiasdiez commented Apr 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

user202729 commented Apr 19, 2025

Uh oh!

user202729 Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

tobiasdiez Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobiasdiez commented May 17, 2025

Uh oh!

dimpase commented May 17, 2025

Uh oh!

tobiasdiez commented May 17, 2025

Uh oh!

dimpase commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobiasdiez commented May 19, 2025

Uh oh!

user202729 commented May 19, 2025

Uh oh!

tobiasdiez commented May 19, 2025

Uh oh!

dimpase commented May 19, 2025

Uh oh!

user202729 commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

user202729 commented May 19, 2025

Uh oh!

tobiasdiez commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

user202729 commented May 20, 2025

Uh oh!

user202729 commented May 24, 2025

Uh oh!

tobiasdiez commented May 27, 2025

Uh oh!

vbraun commented May 28, 2025

Uh oh!

dimpase commented May 29, 2025

Uh oh!

user202729 commented May 29, 2025

Uh oh!

vbraun commented May 29, 2025

Uh oh!

dimpase commented May 29, 2025

Uh oh!

dimpase commented May 29, 2025

Uh oh!

Uh oh!

tobiasdiez commented Apr 19, 2025 •

edited

Loading

github-actions bot commented May 17, 2025 •

edited

Loading

dimpase commented May 17, 2025 •

edited

Loading

user202729 commented May 19, 2025 •

edited

Loading

tobiasdiez commented May 20, 2025 •

edited

Loading